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INTRODUCTION 

The great majority of water and wastewater plants use tra- 
ditional digital and analog (4-20 mA) input/output (I/O). 
Fieldbuses have not been widely accepted in the water indus- 
try until recent years. Only a handful of water plants in the 
United States use fieldbus technology extensively. 

Fieldbuses such as HART Networks, Prohbus, and 
ControlNet have seen good acceptance in water distribution 
systems as a way to control held operations and calibrations 
of process transmitters. Also, multi-variable transmitters are 
often used fieldbuses to simply access the device data. 

Wireless instruments have been very slow to gain accep- 
tance in the water industry in local operations. However, it is 
more common to encounter groups of instruments that con- 
nect to a local remote I/O panel which communicates with a 
central controller either through fiber optic, data highway, or 
wireless. 

In this chapter, telemetry and control of water treatment 
plants (WTPs) and waster water treatment plants (WWTPs) 
will be explained referring to a particular application exam- 
ple in the United States. 

EXAMPLE APPLICATION OF WATER TREATMENT PLANT 
Using Profibus 

Among other choices, this particular WTP decided to use 
Prohbus networking for its instrumentation. Since reliability 
was of utmost importance, a redundant architecture had was 
selected. One of the benehts of a network based on heldbus 
technology is the reduction of conduits and wiring needs 
required for the system. Unfortunately, in this plant, the deci- 
sion to use Prohbus was made after a conventional design 
had already been completed. Consequently, the Prohbus was 
a retroht and used conduit that already existed. 

The implementation proved to be difficult because this 
was done in the early years of 2000 and many manufacturers 
were still preparing up to make Prohbus available on their 
instruments. The requirement of a redundant Prohbus con- 
nection made matters much more difficult. 


Although there were some initial startup issues, the 
instrument network was successfully commissioned and at 
the moment is functioning quite well. 

Using DeviceNet 

The instrumentation Prohbus network worked alongside the 
DeviceNet system used for the motor control centers (MCC) 
and variable frequency drive (VFD). The MCC manufactures 
have made built-in I/O for many years that only require a com- 
munications cable. But this has been very slowly accepted 
in the water industry for many reasons. For example, at the 
engineering level, the electrical department and the automa- 
tion departments may have difficult time coordinating smart 
MCCs because this has traditionally been specihed by the 
electrical department. At the held, the electrical contractor 
usually purchases and installs the MCCs but smart MCCs 
require some programmable logic controllers (PLC) exper- 
tise, which is usually provided by separate contractors. 

One of the concerns addressed in the project was the risk 
of electrical noise interfering with communications. It was 
decided to use a hber-optic converter for the VFDs, to pro- 
vide complete noise immunity for communications. 

OVERVIEW OF TELEMETRY APPLICATIONS 

Because water and wastewater systems cover large geo- 
graphic areas, telemetry is used extensively. This will be 
explained in detail. 

Wastewater Liftstations 

A liftstation telemetry system usually covers from 20 to 600 
sites. A typical example of a remote site location is illus- 
trated in Figure 58.1. Telemetry systems send the critical 
information regarding local power, pump operations, reser- 
voirs, and tank overflows. Back to a central control room. 
Information used for maintenance includes run hours per 
pump per day and pump starts and stops. Intrusion is also a 
monitored value. Using telemetry also helps preventing the 
overflows. 
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FIG. 58.1 

A typical remote site location. 

Water Well Fields, Reservoirs, Tanks, and Pump Stations 

Water systems use telemetry to monitor reservoirs, tanks, 
and well fields. Water levels are tracked to prevent removing 
too much groundwater from any given well. Reservoirs are 
monitored to aid in decision making of water intake for the 
best quality water. 

On the distribution side, telemetry sends information 
on the tank levels and pressures back to the water plant so 
that water production can be managed intelligently. Tanks 
are often filled at night when power is less expensive and 
partially emptied during the peak demand of daytime. Some 
systems send information on pressures and flows, knowing 
both of these values can help locate water main breaks. A 
“break” will be indicated by an area of high flow and low 
pressure. 

Systems that have separate pressure zones often use 
telemetry to send the receiving tank level to the booster 
pumps so that they can fill the tank to the desired level and 
shutdown at the right time to avoid a wasteful water tank 
overflow. 


USE OF THE PACKET RADIO 
Packet Radio 

Water and sewer district (WSD) radio telemetry network 
(RTN) is vital to the operation of the department water and 
sewerage systems. The radio network was initially deployed 
to monitor and collect data as part of the wholesale auto- 
mated meter reading (AMR) system. The AMR system is 
a critical business function as it supports the need to have 
accurate and timely billing and information disseminated to 
water utilities located in the surrounding communities. In 
developing this capability, WSD used UtiliNet packet radios 
to construct a wireless data network comprised of individual 
radios and telecommunication towers arrayed in a grid-like 
fashion over the service area. Over time, additional radios 
and supervisory control data acquisition (SCADA) applica- 
tions have been added to the UtiliNet network in order to 


expand WSD capabilities to monitor and control key compo- 
nents of the department’s water and sewer systems. The net- 
work continues to grow in size, currently having over 1000 
radios online. 

UtiliNet History and Design 

In deciding to automate its wholesale metering operations, 
WSD selected a wireless radio network based on UtiliNet 
packet switching radios, currently a product of Cellnet. 
UtiliNet radios have been sold since 1988 primarily to 
utility companies for AMR, load shedding, and SCADA 
use. UtiliNet is also known by its earlier manufacturers as 
Metricom and Schlumberger. These radios use spread spec- 
trum technology in the 900 MHz band, operating at 100 mW 
and 9600 bits per second (bps). 

Spread spectrum radio technology has become a pop- 
ular communications choice for SCADA systems because 
it overcomes the diminishing supply of available licensed 
radio frequencies and is designed to tolerate interference. 
Originally developed by the military, this technique spreads 
a narrow band signal by hopping over a broader portion of 
the radio frequency band. Each receiver understands the 
predefined hopping pattern of the sender and then assembles 
the data. 

In a packet radio system, each radio uses an omni- 
directional antenna to send data in all directions. Any radio 
can be configured as a “repeater” so that it can forward 
radio packets to others. The radio specific destinations for 
data are configured in the remote terminal unit (RTU) and 
are added like a “wrapper” to the raw data. The decisions 
on the best transmission path for each data packet through 
the radio network are executed by algorithms in the radios. 
Because the raw data are wrapped and isolated, a variety 
of SCADA systems and RTUs can operate on the UtiliNet 
radio network using multiple protocols simultaneously. A 
distinct advantage of this network is that a packet can travel 
on any one of a number of paths on the way to reaching its 
destination. This allows data to be transmitted more reli- 
ably and reduces single point of failures. 

Radio Communications Networks 

Figure 58.2 conceptualizes how the various project applica- 
tions interface with the UtiliNet Network. The distribution of 
radios and antenna towers creates a communications network 
whereby message transmissions originate at either a remote 
site or at a head-end (or at a WSD operators station before 
passing to that head-end) and are then transmitted over the 
packet radio network before reaching its final destination. 

Radio Location 

Some sites have a high concentration of radios along the 
city’s riverfront. Other radios are more evenly distributed 
across the entire service area. Knowledge of the geographical 
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FIG. 58.2 

System block diagram ofWSD’s major projects using UtiliNet radios. 


TABLE 58.1 

Project Applications 


Application 

No. Radios 

Purpose on UtiliNet 

DWS-805 

285 RTUs 
90 SCADA PCs 

Billing and Meter Reading System (Meter Pits and 
Community PCs — 86 radios) 

DWS-808 

445 

Repeater network 

PC-713 

234 

Monitoring and Control of Wastewater Collection System 
(WWCS) and Treated Water Transmission System (TWTS) 


distribution of the radios can help to understand the direc- 
tion and distance (and thus the time) information packets will 
travel to reach their intended destination. 

Head-End Radios 

The DWS-805, PC-713, and PC -747 projects have separate 
head-end radios at the central services facility (CSF). In 
addition to head-end radios at CSF, the DWS-805 project 
also has head-end radios at the water board building (WBB). 

Project Applications on the UtiliNet Network 

Over time, a number of SCADA applications have been 
added to the UtiliNet network. Each of these project applica- 
tions has an intended purpose as identified in Table 58.1. 

DWS-805 Meter Reading 

The DWS-805 data acquisition project resolved water bill- 
ing disputes by monitoring master water meters serving 90 
wholesale customers. More than 80% ofWSD’s water revenue 
comes through these meters. The DWS-805 wholesale AMR/ 
SCADA system consists of water meters, pressure transduc- 
ers, and other devices which transmit flow, pressure, and 
temperature readings, and other information simultaneously 


to both WSD and the WSD suburban wholesale customers. 
The meter pits are geographically dispersed across a wide 
area, so using a packet radio network is an economical way 
to transmit the information using an unsolicited reporting 
protocol. 

A total of 285 metering sites report meter readings to CSF, 
WBB, and wholesale customer communities. The DWS-805 
project furnished each of the 90 wholesale customer commu- 
nities with radio access. 

Field Sites There are 285 RTUs at the meter pits, each with 
a radio in the cabinet connected to an antenna on a 15 ft pole 
(Figure 58.3). The RTUs monitor flow meter rates and totals, 
as well as pressures up and down stream of the meters. Each 
RTU can send data up to 11 destinations. Typically only 
three destinations are used: CSF or WBB, and one or two 
wholesale customers. In addition, mobile PCs with radios 
are also used occasionally to field calibrate the meters and 
other instrumentation. The metering sites report in every 
5 min to the wholesale customers. Every 5 min, the sites 
send the last data set to the WSD head-end radios at the 
WBB or CSF. This amounts to 288 messages per day per 
site. If needed, delay parameters can be used to stagger the 
report back to the head-end radios. The RTUs store flow- 
meter data up to 31 days in 5 min intervals. When radio 
communication is interrupted, the central historian notices 
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FIG. 58.3 

Each RTU has a radio in a cabinet connected to an antenna on a 
15ft pole. 

the missing data and requests it over the radio network 
during the nightly diagnostic. The stored data can also be 
retrieved directly from the RTU by traveling to the held/site. 
The design requirements call for a minimum of two radio 
paths for each meter pit site. 

The DWS-805 SCADA system has the ability to link-off 
any site remotely. When a site/destination is linked-off, no 
data is transmitted from that site to the linked-off destina- 
tion. When a link-on command is received by the RTU, the 
RTU resumes sending data to the specified site(s). 

Protocols Because of the large number of remote sites, the 
traditional polling or master/slave communication protocol is 
not used. Almost all messages are unsolicited, being either 
time or event driven. The DWS-805 system uses a modified 
Allen-Bradley DF1 protocol with the following features: 

• No Acknowledgment (ACK) for 5-15 min messages 

• “ACK” only sent on configuration changes and excep- 
tion reports 

• Immediate reports upon any exception 

• Incorporates latitude, longitude, and color information 
into the DF1 RTU packet 

• Configuration changes/updates and diagnostic reports 
and historical data requests/replies 

DWS-808 Backbone 

In exchange for real-time meter data from their meters, 
customer communities provide space for repeaters on their 
radio towers and other tall structures. These radios are not 
attached to any RTU or PC. Their sole function is repeating 
data from neighboring radios and they are the backbone of 
the network. Radio antennas are installed at high elevations 
to provide a clear path for radio transmission. The backbone 
also includes repeaters mounted on lighting poles at most 
WSD stations and basins and on buildings or stacks at WSD 
plants. Most repeater sites have two radios, each equipped 
with a 6dB omni-directional fiber antenna. One radio is 



FIG. 58.4 

Typical WSD station with two repeater radios on a lighting pole 
(not pictured). 

equipped with a fiberglass antenna and the other radio has 
a smaller “whip” antenna which is mounted about 5 ft lower 
than the other antenna. Repeater radios are programmed to 
send a heartbeat diagnostic message to the DWS-805 head- 
end every 6 h. 

Data Destinations The DWS-805 head-ends are at two sites: 
WBB and CSF. Each WSD head-end has nine antennas/radios, 
eight for communication, and one for diagnostics. Either the 
WBB or CSF head-end can be the primary head-end. These 
head ends are connected to a communications server at each 
site. The communications servers connect to each other 
through Ethernet using a redundant T1 link between the WBB 
and the CSF. 

The wholesale customer flowmeter readings and pres- 
sures are collected and retained long term by a specially 
designed historian at the CSF. Testing and held calibration of 
the meters are performed using held test set PCs, which are 
also equipped to communicate via radio to both WSD and to 
the WSD suburban wholesale customers. 

Communities have PCs and local radios to receive the 
same data that WSD uses for billing. These are typically 
conhgured with two repeater radios (Figure 58.4) on the 
roof and one radio inside the building that is attached to the 
SCADA PC. 

Diagnostics Several levels of diagnostics are implemented 
on this system. The DWS-805/808 radios provide a timed 
ALIVE message on a 6h interval and report to a nightly 
request for diagnostic information, which allows monitoring 
of the DWS-805/808 network on a daily basis. Both radio and 
data transfer diagnostics are used. End sites, repeater sites, 
and wholesale customer sites are checked for proper opera- 
tion. If a site is off for more than 6 h, an alarm is generated. 
Diagnostic reports are run every night at conhgured intervals 
(currently, starting at 1 am). 

Reporting Criteria Sites must report 90% of their data to be 
considered acceptable. If the historical hies have data gaps, 
the historian generates radio requests to retrieve this infor- 
mation from the RTUs. 
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SCADA TELEMETRY INTEGRATION 

The monitoring and control utilizes the PC-713 SCADA sys- 
tem, which consists of Ovation controllers, serial link con- 
trollers, Control Microsystems SCADAPack RTUs/PLCs 
with graphic display on PCs. One of the control panels of 
PC-713 is illustrated in Figure 58.5. It utilizes the packet 
radio network as a part of the overall communication net- 
work. The radio network is used for primary communication 
between the SCC and approximately 200 remote RTU sites. 
The radio network is also used for backup communication 
between the SCC and 36 pump stations. The pump stations 
and the remote RTU sites are spread over approximately 
1000 mi 2 , with the furthest site being more than 50 mi from 
the SCC. 

Field Sites 

There are two different types of sites, Type A RTUs and Type 
B RTUs. 

The 36 Type A sites use Ovation Controllers with SLC 
modules. Currently, 21 Type A sites are active. Control 
Microsystems SCADAPack RTUs are used at the 192 Type B 
RTU sites. Pressure, remote valve operation, and rain gauge 
data are all sent in one data packet. 

The Type B remote sites are configured for three modes 
of operation. The first is a one-way transmission only, from 
the site to the SCC. The WWCS sites in this category are 40 
sewer meters, 7 level sensors, and 31 precipitation gauges. 
The TWTS sites in this category are 40 pressure sites. The 
next mode is where the communication is primarily one- 
way, remote site to SCC, but an adjustment parameter is 
infrequently sent from the SCC to the site. The 60 combined 
sewer overflow (CSO) facilities are in this category. The last 
communication mode is also primarily to report from the site 
to the SCC, but some devices are operated from the SCC, 
typically a valve is opened or closed. In this category are 8 
WWCS valves, 2 WWCS dams, and 11 TWTS valves. 

The 36 pump stations (WWCS and TWTS) are config- 
ured, with the radio network being the backup communica- 
tion to a dedicated digital phone line. When in the backup 
mode, the Ovation Controller is throttled not to send any 



FIG. 58.5 

The PC-713 SCADA system control panel. 


faster than 5 min and a keep-alive signal of 15 min for the 
TWTS stations and 30 min for the WWCS stations. Upon 
failure of the primary communication, the throttled timing is 
reduced to 15 s. The primary mode of communication is from 
the station to the SCC. Control commands (start-stop and 
position) are sent from the SCC to the station. The real time 
monitor and control data are the same for the radio network 
and the phone line; thus other than the speed of response, the 
switch from primary to backup is transparent to the operator. 
Two functions are lost when the station is in radio backup 
mode. One of the functions is the sending of historical data to 
the SCC. These data are buffered at the station and sent when 
the phone line is restored. The other function is a remote 
function that allows an engineer/operator to log into the sta- 
tion, make configuration changes, and view certain values 
not required for control. 

Protocols The basic protocol for the radio network is unso- 
licited report on exception. Analog values are inhibited from 
causing an exception unless they exceed a dead-hand. With 
proper dead-hands, typical sites report an exception every 
10-20 min. The protocol also has a throttling scheme that 
will not allow a site to report an exception any faster than 
the throttling time. This time is typically set at 15 s to 1 and 
5 min for the Type B RTUs. 

Data Destination The PC-713 system has two head-end 
processors with eight radios each for the Type A RTUs and 
the Type B RTUs, for a total of 16 head-end radio, plus one 
diagnostic radio at the CSF. The head-end radios rotate their 
transmit duty and no messages are allowed to queue. 

The primary use of the data is to provide an operator 
with a current real time view of data required to operate the 
transmission systems. These data are continuously displayed 
on computer graphic screens, thus providing the operator 
needed information to control the water system. In addition 
to the current state displays, the data is recorded in a histo- 
rian. From the historian data, various internal and external 
reports are generated to provide information to analyze a 
previous event. 

Diagnostics The system alarms if three consecutive pack- 
ets from any remote site are missed at the SCC. Future diag- 
nostics will include the recording of remaining time-to-live 
(TTL) for each data packet and the recording of remaining 
hops for each data packet. 

Normal Radio Traffic 

In a network of more than 1000 radios, of which approxi- 
mately 500 are directly connected to data producing RTUs, 
there is the potential that a large number of data-packets are 
present on the network. When there are large amounts of data 
transversing the network, more radios are busy routing data 
and fewer radios are available for processing new data. Each 
radio base contributes to the radio traffic. The objective of 


© 2012 by Bela Liptak 



58 Telemetry Control and Management of Water Treatment Plants 863 


this arrangement is to control and manage the flow of data, 
meeting the performance requirements, and avoiding time 
periods where there is too much data on the network at any 
given time. The challenge of this effort is the coordination 
required by each SCADA system to balance the load. 

Flow of Data Report-by-exception is the best transmission 
scheme for handling large amounts of data across the radio 
network. Under this scheme, the remote site RTU has com- 
plete control over the reporting frequency as it sends data 
only when an event (change of state or pre-timed interval) is 
detected. If properly programmed and adjusted, report-by- 
exception greatly reduces the amount of network traffic in 
comparison to continuous reporting which is typically used 
with high bandwidth networks. Sites that do not experience 
large process changes will not send data as often as sites that 
do. Two types of exceptions exist on the network: change of 
state and time-based exceptions. 

A change-of-state exception triggers a data packet each 
time something changes at the site. Change-of-state excep- 
tions include equipment status change, change in an analog 
values, and any command issued from the SCC. To prevent 
continuous reporting of analog values, deadbands are used 
to reduce the reporting frequency. Deadbands are placed on 
analog values so that only significant changes in value are 
reported. This prevents repeated notification for a bouncing 
value, for example, water pressure. Without deadbands, sites 
would continuously send data over the network each time the 
value changes even by an insignificant amount. 

A time based exception (or heart-beat) refers to the report- 
ing of data on a time interval. The 285 RTUs send wholesale 
meter data at pre-defined intervals. Data from each RTU is 
sent at 5 min intervals to wholesale customers and back to 
the DWS-805 head-end at 15 min intervals. Another example 
of time-based reporting is the diagnostics performed every 
6h for the DWS-808 repeaters and the nightly historical data 
update for the DWS-805 system. The PC-713 SCADA project 
also uses time based exception reporting. 

Exception Reporting Issues There are several poten- 
tial issues that need to be addressed while using exception 
reporting. At the beginning of the radio network evaluation, 
the 285 m pit RTUs were all synchronized to send their data 
at exactly the same moment because geographic distribution 
was thought to balance the load. 

To distribute the load more evenly than using geographic 
distribution alone, the meter pit RTUs were configured to 
transmit their data at different offset intervals. Therefore, 
instead of all the RTUs sending data at the same instant every 
5 min, the data can be continuously sent at different times 
throughout the 5 min periods. This coordination was per- 
formed using the delay parameters programmed within each 
RTU. As additional radio projects are added to the network, 
balancing the load of messages should be considered. 

Another potential issue involves the change-of-state 
reporting scheme implemented for PC-713. To combat 


frequently changing conditions, they implemented mini- 
mum and maximum clamping timers to control the report- 
ing frequency. For instance, an erratic sensor bouncing up 
and down will generate rapid data transmissions under the 
traditional change-of-state reporting scheme. The minimum 
timer is used to control and prevent data from flooding onto 
the network. The maximum report timer is used to provide 
the periodic heartbeat function. 

RADIO NETWORK ISSUES 

Starting in February 2005, PC-713 radios and RTUs were 
added to the network and twice the network experienced 
diminished performance as a result of high packet transmis- 
sion rates. Once the packet transmission rates were reduced, 
the DWS-805 meter readings were reported as normal in the 
network operation. However, the network timing abilities 
continued to be questionable for control applications at the 
pump stations. As a result, the backup communications at 
these sites was left on the 56 K phone lines. 

Due to the requirements of the PC-713 radio communica- 
tions and the overall performance of the entire radio commu- 
nications network, a radio study was performed to evaluate 
the radio network, take immediate corrections, and make 
recommendations for long-term corrective measures. The 
following list of major issues had been identified: 

• The delay and variability by which radio messages 
are processed, particularly commands that are sent 
to control the operation of the pumping stations and 
other key components of the water and wastewater 
infrastructure 

• The frequency and quantity of messages that do not 
reach their intended destination, particularly in the 
North 

• The delay in transmission of information packets 

Radio system user interviews revealed that depending on the 
intended application of the PLC/radio, network requirements 
were different. For example, the DWS-805 wholesale meter- 
ing application does not require immediate response; how- 
ever, it requires a network that delivers a high percentage of 
data packets. On the other hand, the PC -713 command-and- 
control application not only requires a network that delivers a 
high percentage of data packets, it also requires the data to be 
delivered and confirmed on a much more stringent time basis. 

Identified Concerns 

At the start of the network evaluation, the radio study proj- 
ect team compiled a list of issues apparently affecting the 
packet radio network and its SCADA systems. These issues 
helped to establish the project’s direction and focus for the 
effort. Several of the issues were only specific to an indi- 
vidual project; however, many of the issues applied for the 
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entire network. The radio network issues were identified as 
follows: 

• Long and varying data packets latencies 

• Lost packets 

• Misaddressed data packets 

• Head-ends decommissioned without stopping site 
transmission 

• RTU network traffic 

• RTUs transmitting to site that were powered-off with- 
out issuing a link-off command 

• Lack of domain configurations 

• Poor connectivity 

• Cul-de-sac of packets 

• Improperly set priorities 

• Excessively long for TTL and hops (Luck) settings 

• Misconfigured radios on the network 

• Obsolete firmware 

NETWORK ROUTING 

The UtiliNet radio network thrives on the ability to dynami- 
cally route data packets through any radio within its mesh 
network to any destination if the terrain conditions are 
nearly ideal. However, WSD has areas in its network where 
the antenna heights and location are compromises between 
available sites, recurring costs, and network performance. 
When routing options are limited, packets are forced to 
travel in certain directions, oftentimes taking the same paths 
to its destination as other packets. The lack of routing options 
inhibits the network’s ability to route data in different ways, 
ultimately causing the network to slow down and become 
congested. 

Specific issues that affect the routing of data on the net- 
work are included in the following: 

• Connectivity 

• Interference 

• Repeater latency 

• Domain routing and non-routers 

• Routing decisions 

• Misconfigured packets 

• Misconfigured radios 

• Old version of firmware 

Connectivity 

Lack of connectivity inhibits each radio’s ability to route 
through a single radio or a series of radios, increasing the 
overall network latency. To combat this problem, increasing 
connectivity helps establish more reliable communication 
through the network by expanding routing options through 
more radios. 

Criteria for “Quality” Radio Connectivity Three key per- 
formance indicators (KPIs) define the quality of the radio 


connectivity between radios on the UtiliNet radio network: 
RSSI, TICK, and DACK. When all three KPIs are met 
between radios, a "quality” communication link is formed. 
This quality communication link provides an excellent com- 
munication that is highly available and repeatable. However 
when KPIs are not met, one may begin to observe a decline 
in a number of network performance measures. A reliable 
UtiliNet radio network should be made up of many quality 
communication links that provide multiple paths to every site 
throughout the entire system. If there are only a few or no 
quality communication links in any single geographical area, 
then communications to and from that area are likely to be 
affected. The KPIs are described in the following. 

Received Signal Strength Indication (RSSI) The RSSI indi- 
cates the relative signal strength from one radio to another. 
The strength of the connection affects the time it takes for a 
data packet to get from one radio to another. If the signal is 
weak, then the transmissions between radios takes longer and 
is more prone to failure. Any transmission failure requires 
a retransmission of data. According to the manufacturer, an 
acceptable RSSI value for communications should be greater 
than 130. Signal strengths around and below 120 are consid- 
ered marginal and should not be considered dependable over 
time. With a strong connection, the manufacturer suggests 
that the quickest time duration between each hop is about 1 s. 
Although test data gathered during the study indicated that 
successful data transmissions can occur at the lower signal 
strengths (120-130), the weaker signal strengths will nega- 
tively affect transmission times. 

Tickle Success Percentage (TICK) The TICK value indicates 
the availability of one radio to another radio. If the value is 
low, then the radio is unavailable or busy communicating 
with other radios. A radio that is not busy will have a higher 
tickle value. This metric is most meaningful when signal 
strengths between radios are strong. A desirable communica- 
tion link should have a TICK average of 50% or higher. This 
means that the radio is available to its neighbors to communi- 
cate more than half of the time. A marginal tickle percentage 
is below 30%. 

Data Success Percentage (DACK) The DACK percentage 
success indicates how successful data transmission is to and 
from a radio after initiating a transmission. A low DACK 
value means that a radio is probably experiencing signal 
interference, possibly from another UtiliNet radio. A desir- 
able communication link should have a DACK of 80% or 
higher; however, a value below 70% is suspicious and may 
require some investigation. 

The combination of KPI values provides an indication 
of the radio network quality. As explained above, a quality 
communication link should deliver the data successfully. 
However, to achieve the highest reliability in data delivery, 
it is recommended that each radio at a station or site have at 
least two "quality” paths. 
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Interference 

Radio interference can significantly affect radio communi- 
cations and routing options available on the network. The 
consequences of interference to the radio system will reduce 
RF connectivity. This is indicated by a reduced tickle suc- 
cess rate and poor data success rate to another radio. The 
Routman software utility produces spectrum reports that are 
useful in identifying RF interference that may cause connec- 
tivity problems. 

In-Band Interference This type of interference is from 
other devices within the 900 MHz frequency range as the 
packet radio system. This type of interference cannot be fil- 
tered out and other measures must be taken to reduce this 
interference. 

In-band interference can be caused by two or more radios 
located at the same site. Several of the repeater sites con- 
tain two radios to increase data handling capacity and to 
provide redundant paths; however, proper antenna spacing is 
required and is typically the best way to correct this type of 
interference. 

Antenna Mounting There are numerous radios that have 
good RSSI and see other radios with high availability (TICK 
percentage); however, they have very low data success rate 
(DACK percentage). According to the manufacturer, this 
seems to indicate interference experienced by the radios. 
Since interference from other radio systems such as the 
microwave systems and cellular phone systems have been 
investigated, potentially, the interference could be from in- 
band interference, resulting from other UtiliNet radios. The 
antenna mounting procedures state that the minimum separa- 
tion of radio antennas should be at least 2Vi ft vertical or 28 ft 
horizontal. Candidates for this type of interference are the 
head-end radios, co-located repeaters, and sites with UtiliNet 
radio antennas of multiple projects. This could explain the 
repeater phenomenon of one radio being able to see many 
radios; however, the other radio does not see nearly as many. 
The intent of the co-located radios is to provide a paral- 
lel radio path to increase bandwidth. Having this situation 
defeats the purpose of having the second repeater. 

Out-of-Band Interference This type of interference is 
caused by other communication systems outside of the 
900 MHz frequency band. Their signals can be so strong that 
it interferes with the packet radios. Typical sources of out-of- 
band interference include cellular base stations and paging 
systems. 

The best way to prevent this type of interference is using 
RF filters. As a general rule, higher sites that are more sus- 
ceptible to outside interference should be equipped with a 
filter. The UtiliNet WAN gate repeater radios (DWS-808) are 
already equipped with an internal filter. All PC -71 3 radios 
use external filters. 


Repeater Latency 

Repeaters having high RSSIs and where the TICK and DACK 
percents are very low may be too busy. Repeaters can handle 
up to 160 radios, although a few can currently handle more. 
Maximum radius setting, currently set at various distances 
(in miles), can be used to reduce the number of other radios 
seen by the repeater. A balance needs to be kept between the 
desired distant communication and the undesired overload 
of local communications. Routing choices can be somewhat 
controlled by 

• Reducing the maximum communication radius 

• Segregating the radios into domains 

Domain Routing and Non-Routers 

Domain routing eliminates undesirable routing choices by 
forcing packets to travel through a core highway of radios 
and not route through areas of poor connectivity. The WSD 
network is divided into core and domain radios. Core radios 
are routing radios that serve as a highway to route packets 
between domains. Domain radios are radios that have con- 
nectivity among radios in their domain but are not good rout- 
ing choices for packets destined to other domains. Radios 
are pre-configured to be routers by default. This means that 
any radio can participate in the network and provide more 
options and paths to route data. Certain radios may be poor 
router candidates because of limited options and poor con- 
nectivity. To improve network routing, change these radios 
from routers to non-routers. 

Over the course of the network evaluation, recommenda- 
tions for new domain assignments and non-routers have been 
implemented. These assignments have proved beneficial to the 
network since packets have entered cul-de-sacs less frequently 
and tracer reports have shown fewer packets getting trapped 
in radio areas without exiting. The main indicator of the more 
appropriate domain and non-router assignment is evidenced 
by the lessened variability of ranges of minimum and maxi- 
mum parameters such as total hops and RT times. Because 
these values are closer together, the network is becoming 
more repeatable and consistent. In order to reduce the number 
of routers in the network, the following actions were taken: 

• The weaker core routers changed to domain routers 

• The weaker domain routers changed to non-routers 

Routing Decisions 

When a data packet is received by a radio and needs to be 
routed to another radio, there is a procedure followed by the 
radio to determine the next radio and send the data packet. 
The procedure is generally described as follows: 

1. Construct a scan list. A list of neighboring radios that 
qualify to pass a packet to. 
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2. Tickle a radio. Getting a radio’s attention to prepare 
it to receive a transmission. If an “acknowledge” is 
received, then the radio sends the data. Otherwise, the 
radio will tickle the next radio within the scan list. 

3. Send the data packet. After a successful tickle, 
the radio will attempt to send the data packet. If an 
“acknowledge” is received, then the sent data was con- 
sidered successful and the process is complete; how- 
ever, if the transmission fails, then the next radio in 
the scan list is tickled and the procedure starts over. 

The most significant step in the routing process is the first 
step, constructing the scan list. The order of the radios on 
the scan list is determined by the best radios (availability, 
direction, and connectivity) to the least desirable. The other 
parameters in the scan list are ordered by each packet’s mood 
(configurable parameter). When a packet is received, the 
radio references the mood and then constructs its scan list 
based on that mood. 

TTL: The TTL setting is a way to remove misconfigured or 
misdirected packets from the network. After a fixed amount 
of time, the packet is automatically killed if it has not reached 
its destination. Undelivered packets reduce system capacity. 

Luck: The Luck setting is another way to address miscon- 
figured and misdirected packets on the network. Similar to 
TTL, Luck is also used to terminate a packet if it has not 
reached its destination as expected. The Luck setting within 
a radio refers to the maximum number of hops that a packet 
can take prior to expiration. If an end radio typically can send 
data over to the head-end with five hops, then a packet that 
travels more than five hops has mostly likely taken a differ- 
ent path of delivery. This change will eliminate “wandering 
packets” from tying up the network. 

Misconfigured Packets 

During January 2006, it was discovered that several sites 
were incorrectly programmed sending packets to sites with 
incorrect addresses. With the move of the SCC to CSF and 
decommissioning of the PC-695/698 head-end radios at the 
WBB, the PC-695/698 PLCs were left sending data packets 
to sites that no longer existed. These misaddressed packets 
can tie up the system until they expire via TTL parameters of 
each data packet. Packets without a real destination must be 
eliminated from the system. 

The DWS-805 sites can be configured to send to a mobile 
test set PC with a radio. If this function is not turned off 
after testing, this could generate orphan messages that stay 
on the network until they run out of TTL. The same is true 
if a wholesale customer radio or community SCADA PC is 
switched off. However, the DWS-805 project now has a pro- 
cess in place to monitor and shut-off these messages. 

High priority packets can cause other packets to be 
blocked in the system. In January, there was a band of net- 
work congestion on the west side of the city. The southwest 


area of metering was reporting delays and communications 
coming from the west were slow. Reports of prolonged delays 
are in reference to the PC -713 radios. The PC-713 sites are 
heavily concentrated along the waterfront. It is this latter line 
that may be creating the artificial barrier for the southwest 
meters. When reports were run in March, there was no spe- 
cific geographic concentration of slow sites. However, the 
PC-695/698 PLCs were observed sending an abundance of 
high-priority messages and would have blocked traffic in this 
manner. 

Misconfigured Radios 

Each radio in the network is programmed with a unique lati- 
tudinal and longitudinal (lat/long) coordinates. The UtiliNet 
radios use these coordinates to calculate and determine 
which direction to pass the data. If these addresses are incor- 
rect, then the radios will try and route the data to radios that 
do not exist where they are reported to be. It is important 
that each radio be addressed correctly to route data across 
the network. 

Cul-de-Sacs Packet tracers have shown “cul-de-sacs” or 
areas of radios that have problems forwarding data into the 
direction of the final destination. A “cul-de-sac” is present 
when a radio does not have forward connectivity to another 
radio, thus having to travel backwards in order to establish a 
new path. Cellnet recommends using only the default setting 
and avoiding the use of advanced settings. As the project has 
evolved, packets traveling into cul-de-sacs have been docu- 
mented and resolved by preventing the packet from entering 
these areas by organizing the radios into their own domain 
and/or reconfiguring the radios as non-routing. This problem 
should be minimized as the domains are redefined within the 
radio network. 

Old Version of Firmware 

The issues identified above are direct contributors to high 
network latency which is currently the main problem affect- 
ing PC-713. By correcting the problem contributors, network 
latencies, and the data, success percentages will improve. 
Two significant contributors to high network latency are 
radio traffic and network routing. 

OTHER ISSUES WITH OVERALL RADIO NETWORK 

Besides radio traffic and network routing, there are also sev- 
eral issues with the overall radio network. 

Radio Ownership 

Throughout the many interviews with multiple parties, total 
ownership of the radio system is very unclear. It appears that 
no one person or group is entirely responsible for the overall 


© 2012 by Bela Liptak 



58 Telemetry Control and Management of Water Treatment Plants 867 


radio operation. This function is critical to the overall coor- 
dination of a uniform radio network. Discussions with IS 
indicate that they are capable of handling this responsibility; 
however, it appears that they are not comfortable with own- 
ing and maintaining the PLCs that supply the radio traffic. 

In addition to owning the radio network, it is apparent 
that a single group needs to be responsible for all configurable 
components of the network including the radio and PLC pro- 
grams. This can provide the ability to standardize functions, 
logic, and parameters required for the system. Additionally, 
procedures for adding and removing radios from the network 
can also be overseen. 

Designate WSD “Owner” of the System 

From the experience gained in this project, it is recommended 
that a mid-to-senior level management person be designated 
radio network “Owner” and be assigned responsibility for: 

1. Regular housekeeping of the radio network: The 
WSD “Owner” should be primarily responsible for 
every radio currently operating on the WSD radio net- 
work. The “Owner” should maintain the master radio 
node -list including all configuration parameters and 
settings required for each radio. It is recommended 
that the “Owner” manage routine radio maintenance 
including regularly scheduled site visits and diagnos- 
tic report analysis. The “Owner” would also maintain 
all documented network standards that should be used 
to drive ongoing and new radio system activities for 
the network. 

2. Coordination for network expansion: The WSD radio 
“Owner” should provide project oversight for all 
efforts interfacing with the WSD radio network. The 
“Owner” should enforce radio practices and standards 
for programming, installation, and testing. It is recom- 
mended that the “Owner” be familiar with the control 
system architectures to coordinate reporting frequen- 
cies, communication protocols, and proper address- 
ing of the radios, and data packets for the functional 
systems. 

3. Coordination within internal departments: The WSD 
“Owner” should be the single point of contact within 
the WSD organization for all radio-related issues. 
Similar issues noted repeatedly throughout personnel 
interviews conveyed lack of coordination and direc- 
tion when relating to the radio network. A single point 
of contact is required for personnel to report problems, 
request work orders, or coordinate site visitation. The 
“Owner” should also notify WSD personnel sched- 
uled work or events that might affect operations and 
users of the system. 

4. Coordination for neighboring utilities: The “Owner” 
should be responsible for coordinating all radio-related 
issues involving any outside utility interfacing with 
the network. This effort should involve strict oversight 


of new radio and PLC installations and periodic radio 
audits to confirm proper operation and compliance to 
published operating procedures. The radio study has 
shown how easily misconfigured radio systems can 
limit/even prevent communications to remote sites. 
This effort is necessary to insure that not only WSD 
data transmissions are successful but also other utili- 
ties’ data are being sent properly over the network. 

ORGANIZATIONAL RECOMMENDATIONS 

Adopting the above mentioned recommendations will lead to 
the optimal performance of the network and increased satis- 
faction of the network’s end-users. Once the radio commu- 
nications network is optimized, proactive management and 
preventive maintenance of the entire network is needed to 
maintain this optimal status. Through clearly assigned roles 
and acceptance of ownership for the entire system, WSD can 
ensure that the network continuously meets the goals and 
objectives of the organization. 

Standards 

It is recommended that WSD develop and implement opera- 
tional standards related to or involving the UtiliNet radio net- 
work. These standards will provide direction to all radio and 
SCADA participants involved with the design, implementa- 
tion, and maintenance of the radio, and SCADA communica- 
tion systems. 

The standards should be handled as “living documents” 
that will be revised and enhanced throughout the network’s 
evolution as more radios and SCADA projects come online. 
It is anticipated that these standards be distributed to engi- 
neers, contractors, and neighboring utilities interfacing with 
the radio system and they will be the subject of internal train- 
ing, where appropriate, to WSD staff. 

Added Diagnostics 

System diagnostics are only being accumulated for the 
805/808 systems. Information relating to the performance of 
individual radios and the network as a whole is very impor- 
tant not only to identify current problems but also to antici- 
pate potential problems in the future. 

It is recommended that radio diagnostics be configured 
to allow for the monitoring of all UtiliNet generated data 
packet transmissions. These diagnostics will provide WSD 
the information necessary to monitor the amount of data 
being placed on the network. 

To support network oversight, a standard approach to 
diagnostics should be developed and implemented for the 
entire system. The actual data collected and reported should 
include all necessary connectivity and latency information to 
sufficiently monitor the radio network and SCADA system 
(types of statistics gathered under this study). It should be 
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stressed that only pertinent diagnostic data be transmitted, 
to not overburden the network with ineffective data. It is also 
recommended that the diagnostics traffic be balanced across 
the reporting spectrum to level the network load. 

Performance Metrics 

A set of “UtiliNet Performance Metrics” should be devel- 
oped, updated, and used to determine if the system is operat- 
ing within the expected range of performance and to assess 
whether implemented changes or practices are having a 
positive or negative impact. Routine collection and “trend 
analysis” of such metrics can provide an early warning of 
impending problems, typically with sufficient time to take 
corrective actions before system dysfunction is reported by 
the end-users. 

Radio Communications Network Maintenance 

It is recommended that routine radio hardware and con- 
figuration maintenance of the entire network, including 
both PC-713 and other radios and repeaters, be performed 
either by dedicated in-house staff or by specialty contractors. 
Network maintenance would include regularly scheduled 
site visits and diagnostic report analysis for network connec- 
tivity, latency, route analysis, and aging equipment. A his- 
torical record of this information should be maintained and 
reviewed periodically to facilitate evaluation of any changes 
in network performance. 

WSD should move the radio network to a new ID (CRC 
ADDR) and that new ID should be kept confidential to only a 
few WSD personnel (not SCADA integrators). 

Local Automatic Controls 

During emergency operations, the operators could benefit 
from having some local automatic control capabilities at crit- 
ical pumping stations and control sites. Therefore, it is rec- 
ommended that limited automatic controls be implemented/ 
activated at such sites to provide local regulation of equip- 
ment during periods when primary communication via the 
fiber network has failed. 

Operational Recommendations 

WSD operators depend on a reliable radio communications 
network for the monitoring and control of designated WSD 
sites. Operator confidence in the communications network is 
paramount to ensuring that operator information requests and 
control instructions are effectively carried-out to completion. 

In addition, it is important to note that the radio network 
use for DWS-805/DWS-808 is primarily to monitor WSD 
sites, where data transmission speed is not critical and gaps 
in data can be re-accessed for historical use. Under most cir- 
cumstances, DWS-805 data coming in from the radio net- 
work is strictly being logged historically for future reports. In 


contrast, the PC-713 project comprises both “monitor only” 
sites and “command-and-control” sites. The real-time con- 
trol sites require not only reliable communications, but also 
require data to be delivered in a timely basis. It is also noted 
that the primary communications for the PC-713 “control” 
sites is through the Opt-E-MAN fiber-optic communica- 
tions network that provides transmission speeds comparable 
to speeds typically seen for in-plant operations. However, 
for these remotely “controlled” sites, the radio communica- 
tions network is intended to be used as backup to the fiber- 
optic system. Since the radio network’s transmission speed 
is significantly slower than that of a fiber-optic network, 
WSD operators need to be aware that the radio network’s 
responsiveness will be less than observed with the fiber-optic 
network. Based on these observations, the following are 
recommended: 

1. Satisfactory operation of the PC-713 systems on the 
network can be achieved, if acceptable time-delays 
for commands and reporting are achieved. However, 
using the UtiliNet radio communications system, it is 
not likely that the originally specified 15 s maximum 
reporting rate can be achieved consistently. Based on 
the latest RT test results, using a 60-120 s maximum 
reporting rate could be achieved on a more consistent 
basis. If operations cannot accept such a reporting 
rate, other means of communications will have to be 
implemented for these sites. 

2. In addition, under proper network operation, the time 
variance for A-sites could be managed so they do not 
exceed 2 min (120 s). 

CONTROLLERS 

Programmable Logic Controllers 

PLCs have become popular for water and wastewater treat- 
ment plants. In this application, Allen Bradley PLCs were 
used (Figure 58.6) for communications. Although PLCs were 
originally developed for the auto industry, they are a good 
match for the computational demands of water treatment. 

PLCs are known to be very reliable with mean time 
between failures on the order of 40 years. However, there 
seems to be a preference for redundant controllers. This may 
be related to the maintenance benefits that come with redun- 
dancy. One of the processors can be turned off for mainte- 
nance, while the other runs the plant. 

Distributed Control 

Some plants use distributed control, other prefer central 
control. Distributed control is somewhat more complicated 
because separate controller programs need to coordinate 
with each other. They also have to be maintained. The advan- 
tage of distributed control is that if one processor fails or the 
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FIG. 58.6 

The Allen Bradley SLC-5/04 PLC used for communication. 

communications faults, the remaining processors can con- 
tinue controlling their areas. 

This is usually decided on a plant by plant basis. 
Sometimes, using distributed processors can provide enough 
system redundancy so that redundant processors are not 
required. 

Protocol Bridges 

When a project includes multiple digital systems, a problem 
often arises on how to enable the system to communicate with 
each other. Sometimes, this is solved at the operator level 
using the human machine interface (HMI). Both systems are 
independently displayed on the HMI and appear seamless to 
the operator. If a control interlock is needed, then a differ- 
ent solution is used: a protocol converter. One of these was 
supplied for the project so that a limited number of interlock 
points were available between different PLC systems. 


IT vs. Controller Networks 

There was a time when the control network and PCs installed 
in a plant were the first any of the staff had seen. Separation 
of networks was not an issue. Now, most plants have IT net- 
works. Keeping the separation between the control network 
and the business networks is an important goal. 

IT generally is charged with supporting the workstations 
used at the plant. That typically includes installation service 
patches, virus updates, and other updates and patches. 

Industrial Ethernet and TCP/IP-Based Systems 

Industrial Ethernet is common to the PLC controller level, 
and some remote I/O, but is not common at the instrument 
level in this application. 

Fiber-Optic Communications and Networks 

Fiber-optic systems are regularly designed for water and 
wastewater plants. A popular configuration is a redundant 
ring for the control system backbone. These require special 
Fiber-optic switches that support either rapid spanning (RSP) 
tree or a similar proprietary scheme. One switch is desig- 
nated a master in the ring. It normally sends signals out of 
one port and ignores the signals arriving at the other port. 
If a break in the ring is detected, the master starts sending 
data out both ports to restore communications. The rings are 
resistant to cable breaks. 

These systems are popular and it is difficult to have 
enough future spares. The spare fibers are used for other net- 
work purposes including voice over IP ( VOIP) (phone and 
intercom), security cameras, LAN, etc. 
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